The handling systems known from the prior art, more particularly industrial robots, e.g. articulated-arm robots, for positioning an object gripped by means of a gripping apparatus in a specific position and alignment in space, have internal measuring systems which can detect the position of the elements of the handling system and thus give information about the position and alignment of the gripping apparatus in space. In this case, a distinction should be drawn between axially related and spatially related coordinate systems. The axially related coordinate systems relate in each case to an axis of the robot and the respective position thereof. The kinematic chain of the individual axes and elements of the robot and the respective positions thereof yields the unique position (position and alignment) of the robot tool, that is to say of the gripping apparatus, at the end of the kinematic chain. However, the position of the gripping apparatus of an industrial robot is preferably described in a spatially related manner by means of the so-called TCP, the tool center point. This is an imaginary reference point situated at a suitable location on the robot tool. In order to describe what position the robot tool is intended to assume, the position of the TCP in space and its rotation are defined. In particular by means of the so-called Denavit-Hartenberg transformation, the robot controller calculates what position the individual robot axes have to assume, such that the robot tool assumes the predetermined position. The position of the gripping apparatus with the TCP thereof preferably relates to the world coordinate system, the spatial coordinate system or the cell coordinate system which, for example, is directly or indirectly related to the base of the first axis, the primary axis, the basic frame or the robot base of the robot and is coupled thereto. The remaining sub-coordinate systems are related to said world coordinate system, spatial coordinate system or cell coordinate system. It goes without saying that said world coordinate system, spatial coordinate system or cell coordinate system need not be an absolute world system; rather, said system can also be subordinate to another system. This therefore involves a coordinate system which forms the superordinate reference system within the process. Said system is usually coupled to the floor of the process factory, of the process space or of the process cell.
Consequently, it is possible to adjust the gripping apparatus including the gripped object to a specific predetermined position by corresponding inputting to the robot controller. The gripped object is therefore positioned by predetermining a position of the gripping apparatus in space. In this case, however, the following two problems, in particular, arise.
Firstly, the measuring system of conventional industrial robots designed for holding heavy objects is not precise enough that the gripping apparatus can assume an exact position in space such as is required in some manufacturing methods. Although the drives of industrial robots are sufficiently precise, the measuring systems thereof are not. By virtue of the kinematic chain, the measurement errors of the individual measurement elements are multiplied. This arises both from the measurement inaccuracy of the individual measurement elements, in particular of the angle measuring devices of an articulated-arm robot, and from the unavoidable elasticity of the robot elements.
Secondly, the position of the object in space does not yet necessarily emerge from the position of the gripping apparatus and thus the position thereof in space, since the object can usually only be gripped within a gripping tolerance. This gripping tolerance is often far above the required positioning accuracy. Consequently, the gripping error, that is to say the relative position of the object with respect to the gripping apparatus, likewise has to be taken into account. Separate measuring systems that no longer belong to the robot, in particular contactless optical measuring systems, are used for this purpose. Only these make it possible to position the object in space with the required accuracy in a specific position.
WO 2007/004983 A1 (Pettersson) discloses a method for the joining welding of workpieces, in particular pressed sheet metal parts or composite metal sheets. The workpieces to be joined together are held by industrial robots and positioned by the latter relative to one another for the mutual welding connection. During the production of the welding connection, the workpieces are held by the industrial robots in the respective positions, such that the relative position of the parts with respect to one another is maintained. The welding is effected by means of a welding robot, for example. A measuring system measures the positions of the workpieces in order to enable the workpieces to be positioned before the welding operation. The measurement is effected continuously, in particular, during the welding operation. The method described makes it possible to dispense with the otherwise conventional workpiece-specific moulds and workpiece receptacles which are complicated to produce and into which the workpieces have to be fixed prior to welding. The industrial robots can be used universally for differently shaped and designed workpieces, since, by detecting the position of the workpieces by means of the measuring system, it is possible to identify and supervise the workpieces and also to effect accurate relative positioning of the parts with respect to one another. Consequently, it is possible to use a single system for different workpieces. It is thus unnecessary to change workpiece receptacles. In accordance with the disclosure, the method described is suitable, in particular, for welding sheet metal parts, primarily in the automotive industry. A laser triangulation method in which previously defined points on the workpiece are measured is mentioned generally as a possible measuring system. For this purpose, by way of example, reflectors are fitted on the workpiece. In accordance with the description, the position of each reflector can be determined by means of a light source and a two-dimensional detector, such that the position and alignment of the workpiece can be detected by means of three such points. The exact construction of the measuring system is not described in greater detail in WO 2007/004983 A1.
U.S. Pat. No. 5,380,978 (Pryor) describes a method for positioning objects, in particular sheet metal parts, in space by means of an industrial robot. Inter alia, cameras having a corresponding stereo basis for the three-dimensional detection of the position of the object in space are used as a measuring system. The cameras are pivotable for adjusting the field of view and, in one specific embodiment, are designed as a theodolite camera, which can also have a laser distance measuring device. In this case, the theodolite described serves as a precise adjusting apparatus for the camera. Similar measuring systems are also described in U.S. Pat. No. 4,851,905 (Pryor) and U.S. Pat. No. 5,706,408 (Pryor).
US 2009/055024 A1 (Kay) describes a robot arm and control system wherein a fixedly aligned 3D scanning apparatus with a limited field of view is directed at a robot arm and at a target object. Both the robot arm and the target object have markings lying in the fixed fixed of view of the 3D scanning apparatus. The relative spatial position between the target object and the robot arm is detected by means of the 3D scanning apparatus. The robot arm is driven by means of the 3D scanning apparatus in such a way that the target object can be gripped by the robot arm. One disadvantage of the system described is that the limited, fixed field of view of the 3D scanning apparatus enables only a very limited distance between the robot arm and the target object, since the markings of both elements must always lie in the field of view. On account of the resultant requisite large field of view and the limited image resolution of a 3D scanning apparatus, only a limited positioning accuracy is possible, since the markings cannot be detected with sufficient accuracy. Since the method described only describes a relative adjustment of a robot arm—not gripping an object—in the direction of an object to be gripped and, consequently, only a relative positional detection of the markings with respect to one another needs to be carried out, it is not necessary to detect the absolute position and alignment of the 3D scanning apparatus in space. A referencing in the form of a determination of the position of the 3D scanning apparatus in the spatial coordinate system or a detection of the alignment in the spatial coordinate system is therefore obviated entirely.
EP 1 345 099 A2 (TECMEDIC) describes a method for determining a positional deviation of a workpiece gripped imprecisely by a gripper of a robot and for mounting the workpiece on an object by means of the robot, comprising a gripper, comprising an image processing system comprising sensors, such as cameras, and comprising a computer. A plurality of cameras which are stationary or are mounted on the robot arm and which are spaced apart from one another each have a fixed field of view, wherein the fields of view of the cameras overlap. Firstly, the position of a calibration object and of a calibration tool is detected. The workpiece is gripped imprecisely by the robot. The workpiece is moved into an actual lead position. From the positional deviation of the workpiece in the actual lead position from the desired lead position, which was determined beforehand with the aid of the calibration tool, an actual vector is calculated, representing the gripping error of the robot. On the basis of this actual vector, the transformations for adjusting the gripper are calculated, wherein the necessary relative movement between workpiece and object is calculated. The adjustment in accordance with the actual vector is effected solely by means of the robot positioning system, wherein it is assumed that the robot can position the workpiece with sufficient accuracy and robot faults are no longer significant. Since the field of view of the cameras is limited and not adjustable, the method can only be employed in a limited spatial region. Although a gripping error of the gripper is detected by means of the method, a possible imprecision in the robot positioning system is not detected. An ideal, very high robot accuracy is assumed. Moreover, the cameras have to be referenced by means of an external referencing system in a world coordinate system.
WO 2005/039836 (ISRA Vision) describes a method for effecting a movement of a handling device with at least one by means of a controller of an actuator. A movement sequence related to an optically detectable object is predetermined for the controller. A control command for the actuator of the handling device is calculated on the basis of the position and/or the movement state of the identified object and the movement sequence related to the object. A corresponding actuating command is output to the actuator to be moved. In other words, the handling device follows an object and is tracked thereto, wherein the object is neither gripped by the handling device nor positioned. The image recording is performed by a stationary camera, or a camera moved concomitantly with the handling device, with a fixed relative field of view.
What most of these systems and methods have in common is that the positions of a plurality of distinguished points on the object are determined by means of contactless photogrammetric coordinate measurements with the aid of image processing systems.
For contactless photogrammetric coordinate measurement at the surface of an object in the near range, from images which reproduce the object from different perspectives, by transforming the image data into an object coordinate system within which the object is to be measured and which is based on the CAD model of the object, for example, the dimensions of the object and the position thereof relative to further objects in the image are deduced. For this purpose, the image data are processed in a data processing unit. The basis of the coordinate calculation is the determination of the relative camera orientations of the images involved.
In this case, as known from the prior art, there is the possibility of recording in a temporally staggered manner from different perspectives that area section of the object surface which is to be measured, by means of a single camera, and of subsequently processing the respective two-dimensional image data by means of an image processing system to form a so-called three-dimensional image. In this case, the pixels of said three-dimensional image are respectively assigned items of depth information, such that each pixel to be examined, in particular all pixels, are assigned 3D image coordinates in an image coordinate system determined from the cameras and the perspectives thereof. Different image processing methods for generating such a three-dimensional image from a plurality of two-dimensional images showing the same scene from different perspectives are known from the prior art.
Furthermore, it is possible, as likewise known from the prior art, instead of the temporally staggered recording of the area section from different perspectives by means of one camera, to carry out substantially simultaneous recording with the aid of a plurality of cameras. This has the advantage that three-dimensional detection of the area section is possible without camera movement, and detection of the respective camera alignments is also obviated, since the cameras can have a fixed relative alignment and distance with respect to one another.
The prior art discloses different 3D image recording devices which are substantially composed of two or three cameras which, in a manner spaced apart from one another, that is to say having a stereo basis, are accommodated in a common housing in a manner fixedly coupled to one another for the purpose of recording a scene from respectively different, but fixed relative perspectives. Since the recorded area section does not necessarily have characteristic image features which enable the images to be processed electronically, markings can be applied on the area section. Said markings can be generated by means of a structured light beam, more particularly laser beam, which is projected onto the area section by the 3D image recording unit and which projects, for example, an optical raster or an optical marking cross. Such 3D image recording units regularly also comprise an image processing device, which derives a three-dimensional image from the plurality of substantially simultaneously recorded images of different perspectives.
Such 3D image recording units are, for example, the image recording systems from “CogniTens” that are known by the trade names “Optigo” and “OptiCell” and contain three cameras arranged in an isosceles triangle, and the system “Advent” from “ActiCM” comprising two high-resolution CCD cameras arranged alongside one another and also a projector for projecting structured light onto the section to be recorded.
The coordinates of recorded image elements to be measured are generally determined by means of referenced markings within the image, from which markings the actual 3D coordinate measurement takes place. In this case the image coordinate system which relates to the recorded three-dimensional image and is thus related to the 3D image recording unit is transformed into the object coordinate system within which the object is to be measured and which is based on the CAD model of the object, for example. The transformation takes place on the basis of recorded reference markings whose positions in the object coordinate system are known. Accuracies of below 0.5 mm are achieved in this case with the 3D image recording units known from the prior art.
Furthermore, 3D scanning systems in particular in the form of 3D scanners with electro-optical distance measurement are known, which carry out depth scanning within an area region and generate a point cloud. In this case, a distinction should be drawn between serial systems, in which a point-like measurement beam scans an area point by point, parallel systems, in which a line-like measurement beam scans an area line by line, and fully parallel systems, which scan a multiplicity of points within an area region simultaneously and thus carry out a depth recording of the area region. What all these systems generally have in common is that the depth scanning is effected by means of at least one distance measuring beam directed at the area and/or moved over the area. Primarily serial systems are in widespread use and commercially available for example under the product designations “Leica HDS 6000”, “Leica ScanStation 2”, “Trimble GX 3D Scanner”, “Zoller+Fröhlich IMAGER 5003” and “Zoller+Fröhlich IMAGER 5006”. Examples of further systems that may be mentioned include “3rdTech DeltaSphere-3000IR”, “Basis Software Surphaser 25HSX”, “Basis Software Surphaser 25HS”, “Callidus precision systems CPW 8000”, “Callidus precision systems CP 3200”, “Faro Europe LS 420”, “Faro Europe LS 880”, “I-Site 4400-LR”, “I-Site 4400-CR”, “Optech ILRIS-3DER”, “Optech ILRIS-3D”, “Riegl Laser Measurement Systems LMS-Z420i/LMS-Z390i”, “Riegl Laser Measurement Systems LPM-321” and “Trimble VX”.
Moreover, RIM cameras exist, also called RIMs or Range Imaging Systems, which enable image recording of an object with simultaneous detection of an item of depth information for each pixel or for a group of pixels. Consequently, it is possible, by means of a single apparatus, to record a three-dimensional image in which each pixel or a multiplicity of pixel groups is assigned an item of depth information, that is to say distance information with respect to the camera.
A problem of every 3D image recording unit is the limited recording range—owing to design constraints—within which image recording can be effected with the required resolution. During the three-dimensional detection of relatively large objects, effecting a plurality of individual three-dimensional recordings from different positions and alignments of the 3D image recording unit is therefore unavoidable. This multiplicity of relatively small image recordings is subsequently joined together to form a larger three-dimensional overall image by means of compensation of overlapping image regions and with the aid of markings within the recorded area section. Different methods for solving this problem are known from the prior art. One general problem in these methods is that the individual three-dimensional images which are intended to be joined together to form a larger image have to have an overlap region. The discrete altering of the position of the 3D image recording unit from a first area section having at least one reference point to a second area section that is at a distance from the first area section and contains no reference point is not possible by means of the image processing systems if further images linking the two area sections were not recorded. It is therefore necessary to carry out a multiplicity of intermediate image recordings in order to optically link the two spaced-apart area sections to be measured and to enable continuous image processing. The recording of a multiplicity of three-dimensional images having no direct measurement content slows down the entire measurement method and takes up storage and computational resources. Furthermore, the coordinate measurements—which are inevitably beset by small measurement errors—within the image recording during the assembly of the multiplicity of images have a dramatic effect on the measurement accuracy, particularly in the case of distant reference points.
The use of a multiplicity of reference points having known positions in the object coordinate system is therefore unavoidable on account of the limited field of view of the cameras. One advantage of the purely photogrammetric systems described is that the absolute position and alignment of the individual cameras of the 3D image recording unit in the object coordinate system does not have to be determined because the absolute position determination of the recorded pixels is effected from the knowledge of the position of the likewise recorded reference points in the image, the relative alignment of the cameras with respect to one another and the relative positions—calculated by means of triangulation—of the points to be measured relative to the reference points in the image. The measurement system can thus be limited to image-calibrated cameras, the relative position of which with respect to one another is known, and an image processing device.
One disadvantage of all these systems is that, on account of the limited field of view of the cameras and the limited image resolution, adjustment of the field of view either by pivoting or altering the position of the cameras or of the object to be measured is often unavoidable. This is the case primarily when measuring relatively large objects to be measured highly precisely, since a specific distance between the cameras and the object must not be overshot on account of the limited image resolution in order to comply with the required measurement accuracy, but the field of view of the camera given such proximity to the object only permits part of the object to be recorded. Consequently, it is either necessary to use a multiplicity of reference points, such that, during each image recording, a corresponding number of reference points, preferably at least three reference points, lie in the field of view, or it is necessary to have recourse to the positions of object points already determined beforehand, in particular markings on the object.
In this case, as described above, a plurality of individual three-dimensional recordings from different positions and alignments of the 3D image recording unit are effected. This multiplicity of the relatively small image recordings is subsequently joined together to form a larger three-dimensional overall image by means of compensation of overlapping image regions and with the aid of markings within the recorded area section. This costs time and requires the use of markings that are not to be measured per se.
Furthermore, measuring systems and methods are known from the prior art in which the 3D image recording unit is carried by the head of an industrial robot or a portal coordinate measuring machine and is adjustable. On account of the high weight of a high-quality and high-resolution 3D image recording unit, which in some instances is greater than 10 kilograms, precise detection of the position of the 3D image recording unit with the required accuracy, which is equivalent to the image recording accuracy, is not possible since this would require such a stable construction of the handling system that the area of use of the 3D image recording unit would be limited to stationary systems. On account of their comparatively low measurement accuracy, which is considerably lower than that of a precise 3D image recording unit, industrial robots are unsuitable for external referencing. Portal coordinate measuring machines are in turn not designed for carrying heavy loads and, in the case of high mechanical loading, do not yield measurement results that can be used for referencing. For this reason, the position measurement values which are possibly supplied by the handling system and which might give information about the absolute and/or relative position of the 3D image recording unit cannot be utilized for referencing the image recordings, in particular a plurality of three-dimensional image recordings of different, non-contiguous area section.
Although the measuring systems described are also suitable for highly precisely positioning objects in space by means of handling systems and are also used for this purpose, the systems previously known from the prior art are beset by numerous disadvantages. On account of the above-described measuring method effected substantially purely by means of image processing, the methods require a relatively long time and necessitate detecting reference or auxiliary markings that are not to be measured per se. On account of the limited field of view of the cameras, the 3D image recording units are usually arranged in direct proximity to the process, generally on a robot arm or at a small distance from the object. Owing to the associated proximity to the process, the 3D image recording unit is exposed to possible particles and thermal influences which arise as a result of the process—for example during welding. Likewise on account of the proximity to the process, further handling systems have to be coordinated with the handling system of the 3D image recording unit in order to avoid collisions. Movement of the 3D image recording unit and the associated new referencing require a comparatively long time and slow down the entire process sequence. Consequently, an arrangement of the 3D image recording unit remote from the process has been entirely dispensed with hereto for.
The two aims, firstly using a highly precise, contactless 3D measuring system having an accuracy of preferably below 0.1 mm for highly precisely positioning objects by means of industrial robots, secondly a measuring system which is not directly exposed to the process, is flexible to handle, is intended to detect a largest possible action and movement space, and, in particular, can be positioned freely, therefore constitute a conflict of targets that has not been adequately resolved heretofor in the field of industrial object positioning by means of industrial robots.